Analyzing Transferable Knowledge by Pretraining with Artificial Language
نویسندگان
چکیده
本論文では,ニューラルネットエンコーダが学習する知識のうち,どのような構造的知識が自然言語のタスクを解くのに転移可能かを調査する.提案するアプローチでは,自然言語の構造を模したいくつかの「人工言語」を用いてエンコーダを訓練し,そのエンコーダの自然言語の下流タスクにおける性能を評価することで,事前学習データに含まれている構造的知識の転移可能性を計測する.実験の結果,転移可能なエンコーダを獲得するにあたって,事前学習のデータ系列中において,統計的依存関係が重要であること,係り受け関係を持つ際に入れ子構造が有用であることなどが明らかとなった.こうした結果は,エンコーダが転移可能な抽象的な知識として,位置を考慮したトークンの文脈依存性があることを示唆している.
منابع مشابه
On the Effective Use of Pretraining for Natural Language Inference
Neural networks have excelled at many NLP tasks, but there remain open questions about the performance of pretrained distributed word representations and their interaction with weight initialization and other hyperparameters. We address these questions empirically using attention-based sequence-tosequence models for natural language inference (NLI). Specifically, we compare three types of embed...
متن کاملAffordances as Transferable Knowledge for Planning Agents
Robotic agents often map perceptual input to simplified representations that do not reflect the complexity and richness of the world. This simplification is due in large part to the limitations of planning algorithms, which fail in large stochastic state spaces on account of the well-known “curse of dimensionality.” Existing approaches to address this problem fail to prevent autonomous agents f...
متن کاملAnalyzing the tradition, hadith, of self-knowledge with regard to the knowledge of God
Deep thinking on the tradition and analyzing other tradition dealing with knowledge of human soul distinguish this proof from those of philosophers and theologians. Availability, unity of the path and the follower and, most particularly, its being intuitive enter the follower to an objective state and, contrary to conceptual proofs, leads him/her to the presence of God. Again, this traditio...
متن کاملThe acquisition of morphological knowledge investigated through artificial language learning.
Affix knowledge plays an important role in visual word recognition, but little is known about how it is acquired. The authors present a new method of investigating the acquisition of affixes in which participants are trained on novel affixes presented in novel word contexts (e.g., sleepnept). Experiment 1 investigated the role of semantic information on affix acquisition by comparing a form-lea...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: Shizen gengo shori
سال: 2023
ISSN: ['1340-7619', '2185-8314']
DOI: https://doi.org/10.5715/jnlp.30.664